RESEARCH PAPERS 


INFLUENCE OF LEARNING MANAGEMENT SYSTEMS SELF- 
EFFICACY ON E-LEARNING PERFORMANCE 


By 

FLORENCE MARTIN * JEREMY I. TUTTY ** YUYAN SU *** 

* Assistant Professor, Dept, of Instructional Technology, University of North Carolina. 

** Senior Instructional Technology Consultant, Depaul University, Chicago, IL. 

*** Instructional Designer, University of Phoenix, Phoenix, AZ. 

ABSTRACT 

Recent advancements in technology have changed the way educators teach and students learn (Wells, Fieger & 
Lange, 2005). In the last decade, educational trends have progressed towards online and blended instruction. One key 
in this revolution is the development of the Learning Management System (LMS); software that enables the management 
and delivery of learning content and resources to students providing students the flexibility for"anytime" and " anywhere" 
learning. 

Research indicates learner self-efficacy with LMS may be a critical factor in e-learner satisfaction (Lee and Hwang, 2007] 
and performance. The goal of this study was to develop and validate an instrument that measures students' confidence 
with LMS, and explore the relationship between LMS self-efficacy and course performance for e-learners. This study was 
conducted with 68 students enrolled in an instructional technology course. 

Student confidence for accessing the course content, tests and grades, asynchronous communication, synchronous 
communication and using advanced tools were measured. Factor and post-hoc analysis were used to examine 
instrument dimensionality. The complete paper will discuss the full results of the study and designs for further validation, 
particularly with regard to implications for measuring studenf self-efficacy with LMS technologies. Patterns of 
confidence and performance will also be reported and discussed. 
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INTRODUCTION 

Recent advancements in technology have changed the 
way educators teach and students learn (Wells, Fieger & 
Lange, 2005). In the last decade, educational trends 
have progressed rapidly in a movement towards web- 
based instruction and blended instruction. The 
breakthrough of the Internet and other new technologies 
has demanded changes on traditional campuses. The 
conventional ways in which teachers teach and students 
learn have been altered (Wells, Fieger, & de Lange, 2005). 
Online courses have proliferated across schools 
worldwide. Students have the flexibility to take classes in 
the luxury of their own home and at their convenience. A 
Sloan consortium survey reported that 3.5 million students 
took at least one online course during the fall 2006 term 
and there was a 9.7 percent growth rate for online 
enrollments that far exceeds the 1.5 percent growth of the 


overall higher education student population (Allen & 
Seaman, 2007). 

The breakthroughs in technology and increase in online 
enrollment have led to the development of the Learning 
Management System (LMS), Course Management 
System (CMS), and Virtual Learning Environment (VLE) that 
facilitate teaching and learning outside the physical 
classroom. Many universities use a LMS or a CMS to deliver 
their courses. These learning environments can be used to 
totally replace face-to-face teaching in a physical 
classroom, partially replace face-to-face teaching, or 
supplement existing face-to-face teaching (Arbaugh & 
Duray, 2002). The LMS can facilitate learning through 
efficient access to learning materials, providing 
immediate feedback to students through online 
assessments (Breen, Cohen, and Chang, 2003) and 
improved communication between students and 
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instructors through discussion forums and e-mail (Beard 
and Harper, 2002). 

Learning Management Systems 

The North American Council for Online Learning (NACOL) 
who are now known as INACOL (International Association 
for K-l 2 online learning) define the course management 
system (CMS) as, "the technology platform used to deliver 
online learning" (p. 10). With some debate, the terms CMS 
and LMS are used interchangeably by online educators. A 
typcial LMS is software for the creation and editing of 
course content, communication tools, assessment tools, 
and other features designed to enhance access and 
ease of use. 

An LMS enables the management and delivery of 
learning content and resources to students. It provides an 
opportunity to maintain interaction between the instructor 
and students, and to evaluate the students by providing 
immediate feedback on online assessments. Most LMS 
software is Web-based to facilitate "anytime, anywhere" 
access to learning contentand administration. 

Common LMS software used in higher education fall 
undertwo broad categories: 

• commercial systems (e.g.. Blackboard, WebCT, 
eCollege, Desire2Learn)and 

• open-source products (e.g., Moodle, Sakai, Segue, 
Coursework). 

These systems share several essential characteristics 
including 

• High availability - Accessible to diverse users 
(instructors, students & administrators) 

• Scalability - Expandable and Up gradeable to meet 
demand 

• Security - Selectively limit and control access 

• Usability - Convenient and practicable for use 

• Interoperability - Able to work with parts of other 
systems and 

• Stability - Reliable and able to endure load changes 
(Hall, 2003) 

The use of LMS has increased dramatically over the last 
decade. Many colleges and higher education institutions 


have adopted the use of learning management systems. 
In the recent years, K-l 2 institutions, corporate training 
groups have also adopted the use of LMS. More and more 
faculty members are using learning management 
systems to teach their courses. They are able to distribute 
course material easily, make students discuss online 
asynchronously, collect assignments online and post 
grades online. Faculty use learning management 
systems for online teaching, and also supplement it with 
their face to face instruction in their blended courses. 
From a university administrator's perspective, a real 
advantage of online instruction is the ability to enroll more 
students without having to build more classrooms (Bonk & 
Dennen, 2003). 

Blackboard 

Blackboard is one of the leading commercial LMS (or 
CMS) products used in North America and Europe (Munoz 
and Van Duzer, 2005). Blackboard has powerful 
capabilities in three key areas: instruction, 
communication, and assessment. It is the most widely 
adopted learning management system among United 
States post secondary institutions. Blackboard provides for 
a password-protected community where students 
access their courses in an online environment. It has the 
necessary admin iterative tools to make teaching online 
easier (Lowe, 2003) (Figure 1). 

Researchers have compared LMS software based on their 
functionality, user-friendliness, and cost. Some of the key 
features that are evaluated in a LMS are its usability, 
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availability, security, stability, interoperability, and 
scalability (Hall, 2003). Blackboard was the learning 
management system used in this study. 

Figures 1 -5 provide a look at the common Blackboard 
student interface and features used in the courses 
examined in this study. 

Asynchronous and synchronous components 

These new technologies allow for a powerful combination 
of highly interactive instructional material with two-way 
asynchronous and synchronous communication 
between teachers and students. The "asynchronous and 
synchronous" terminology has been borrowed from 
digital communications. Asynchronous means a process 
of transmitting data where each character is transmitted 
separately with no time synchronization between sending 
and receiving devices; synchronous means a type of 
transmission in which the transmission and reception of all 
data is synchronized by a common clock. 

In the online course setting, "Asynchronous" literally 
means "not at the same time." An asynchronous course is 
one in which the instruction is delivered at one time and 
the work can be done at a different time. The advantages 
of asynchronous courses are that it makes it possible for 
"anytime" and "anywhere" learning. Some of the key 
components of asynchronous courses are discussion 
boards, using a learning management system for course 
delivery and transfer of files, and online quizzes. 
Synchronous means "events which occur at the same 
time." Synchronous courses are those in which the faculty 
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and students can interact with one another in real time. 
The advantage of synchronous meetings is that the 
instructor can provide instant feedback on a student's 
performance, and allows the teaching to adapt to the 
students 1 needs immediately even though they are at 
different locations. 

A course could be entirely asynchronous or could include 
both asynchronous and synchronous components. While 
discussion boards, emails, recorded lectures are 
asynchronous, virtual classrooms and online chats form 
the synchronous part of the course. Online courses and 
learning management systems have received a great 
deal of attention in the last few years. In this study, the 
authors examined both asynchronous and synchronous 
components of a learning management system and the 
self-efficacy of instructional technology students in using 
these differenttools, 

Online, Hybrid, and Blended Learning 
Within the online learning community the terms hybrid and 
blended learning are used with little or no difference in 
meaning among educators (Watson, 2009). In general, 
hybrid or blended learning combines online and face-to- 
face delivery of educational content. Allen and Seaman 
(2008) on behalf of the Sloan consortium define online 
courses as, "those in which at least 80 percent of the 
course content is delivered online" (p. 4). The definitions of 
hybrid learning span a continuum from a qualitative 
socializing pedagogy (Dziuban, Hartman & Moskal, 2004) 
to quantitative, Dyjur (2008) defined hybrid learning as: 

The integration of face-to-face and online learning to 
help enhance the classroom experience and extend 
learning through the innovative use of information and 
communications technology, Blended strategies 
enhance student engagement and learning through 
online activities to the course curriculum, and improve 
effectiveness and efficiencies by reducing lecture time 
(para 4). 

The Sloan Consortium defines a hybrid course as having 
between 30 percent and 80 percent of the course 
content delivered online. In this study the quantitative 
definition of hybrid or blended learning is used (Allen & 


Seaman, 2008). 

Many scholars have compared online and hybrid 
learning. A recent meta-analysis by Means, Toyoma, 
Murphy, Bakia, & Jones (2009), examining three studies on 
different courses further reinforced the No Significant 
Difference Phenomenon (Russell, 2001). They found no 
significant difference in student performance between 
online and hybrid courses. Exceptions do exist (Keefe, 
2003; Porier & Feldman, 2004; Campbell, et al., 2008). 
Poirier and Feldman (2004) and Campbell et al. (2008) 
found a significant effect favoring online students when 
comparing an online course with a hybrid course. Keefe 
(2003) found a significant effect favoring performance for 
students in a hybrid course over those in an online course. 
Findings such as these lead hybrid course performance 
and us to examine commonalities and factors that may 
contribute toonline. 

Self-Efficacy 

Self-efficacy is students 1 judgment of their own 
capabilities for a specific learning outcome. Bandura 
(1997) in his self-efficacy theory, defined self-efficacy as 
beliefs in one's abilities to carry out a desired course of 
action. According to Bandura, there are four sources of 
self-efficacy: the self-beliefs of students are formed from 
mastery experience (performance on previous similar 
tasks); vicarious experience (modeling, or the observation 
of others' performance on similar tasks); verbal persuasion 
(feedback from significant others); and physiological and 
emotional reactions (e.g., anxiety) to specific tasks. 
Although the informal term confidence is sometimes 
used as a synonym for self-efficacy, it fails to capture the 
specificity and theoretical base of the construct of self- 
efficacy (Bandura, 1997). 

Furthermore, Bandura suggests the formation of self- 
efficacy beliefs is based primarily on reflection and 
interpretation of past performance (also referred to as 
enactive mastery experiences). Previous experiences in 
which a particular performance was enacted by an 
individual and was perceived by that individual as 
successful will tend to raise self-efficacy beliefs related to 
this performance; those experiences perceived as 
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unsuccessful will tend to lower self-efficacy beliefs. With 
the technological advancement in this decade, it is 
important that students are successful in achieving 
learning and performance outcomes, which results in an 
increase in their self-efficacy beliefs. 

Self-efficacy for technology use may be an important 
factor for student participation and performance. With 
courses being taught fully online and in hybrid settings 
(face-to-face and online), it has become important for 
students to be confident in their technology skills. 
According to Eachus and Cassidy (2002), self-efficacy is 
an important factor in understanding the frequency and 
success with which individuals use computers. Compeau, 
Higgins, and Huff (1999) tested the influence of computer 
self-efficacy beliefs, outcome expectations, effect, and 
anxiety on computer use and found that computer self- 
efficacy beliefs had a significant positive influence on 
computer use. 

Self-efficacy is often measured by self-reported surveys. 
Saade and Kira (2009) developed a survey instrument to 
capture student perceived computer anxiety and self- 
efficacy of using an LMS. The survey contains 18 Likert-type 
questions measuring student perceptions on three 
constructs. The findings indicated that computer self- 
efficacy has a significant impact on computer anxiety 
and perceived ease of computer use. 

Purpose 

LMS self-efficacy, defined as self-assessmenf regarding 
one's skills using a LMS, may be a critical factor in e-learner 
satisfaction (Lee and Hwang, 2007). The goal of this study 
was to develop and validate an instrument that measures 
students' confidence with LMS, and explore the 
relationship between LMS self-efficacy and course 
performance for e-learners. 

Method 

LMSES 

The Learning Management Self-Efficacy Survey (LMSES) 
was initially designed with five LMS-related categories 
(Accessing the course content, Tests and Grades, 
Asynchronous Communication, Synchronous 
Communication and Advanced Tools). Participants were 


asked to rate the items on the survey on a four-point Likert 
scale ranging from (1) Not Confident at to (4) Very 
Confident. 

Participants 

This study was conducted with 68 students enrolled in one 
of two sections of an instructional technology course. 
Thirty-three students were enrolled in an online course 
where a LMS was used for course delivery, and 35 students 
were enrolled in a hybrid environment in which the LMS 
was used as a supplement to face-to-face instruction. The 
study participants were predominantly female (64%) 
below the age of 24 (50%). 

Results 

Five-category grouping before differential analysis 

Student confidence for accessing the course content was 
(M = 2.69), tests and grades (M=2.73), asynchronous 
communication (M = 2.02), synchronous communication 
(M = 1.84), and advanced tools (M = 1.91). The reliability of 
this administration was .96. 

Three different analyses were conducted to test for 
differences between the online and hybrid learners, A 
multivariate analysis of variance (MANOVA) conducted 
on the overall data indicated students enrolled in the 
online course reported significantly greater self-efficacy 
than students enrolled in the hybrid course, F (1, 68) = 
14.194, p <.01. Follow-up univariate analyses for the five 
categories revealed significant differences at the p<.01 
level for four of the five categories, all indicating greater 
self-efficacy for students in the online course. Tukey post 
hoc analyses reveled significant differences on 31 of the 
48 items again, all favoring students in the online course. 
No significant differences were found for items in the Tests 
and Grades category (Table 1). 

Four-category grouping after differential analysis 
Following the first administration of fhe LMSES, differential 
item function analysis lead to the reduction of the LMSES 
from 48 to 24 items (Table 2). From this analysis four factors 
emerged: i) Accessing information, ii) Posting information, 
iii) File management, and iv) Advanced features. These 
four factors combined to account for 91% of the item 
variance. 
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LMSES Functionality before differential analysis, 


Online, 

Hybrid, 

Overall Mean 

Part 1 Accessing the Course Content 





1 would feel confident to .... 





1. Log in to my course in the LMS 


2.88 

2.75 

2.81 

2. Read the text-based announcements posted by my instructor 


2.94 

2.72 

2.83* 

3. Listen to the voice-based announcements posted by my instructor 

4. View my instructor's information, such as name, office hours, and 


2.79 

2.08 

2.42* 

office location 


2.97 

2.78 

2.87* 

5. View the course documents online 


2.97 

2.78 

2.87* 

6. Download the course documents to my computer 


2.91 , 

2.75 , 

2.83 

7. Access the links to the Web resources 


2.94 , 

2.83 , 

2.88 

8. Access the course calendar and tasks assigned 


2.88 

2.83 

2.85 

9. Create a homepage with personal information 


2.73 

1.36 

2.01* 

10. View profiles of other participants in the course 


2.76 

2.22 

2.48* 


Mean 

2.88 

2.51 

2.69* 

Part II Tests and Grades 

1 would feel confident to .... 





11. Take a test/quiz online 


2.67 

2.47 

2.57 

12. View the feedback for the online test/quiz 


2.79 

2.75 

2.77 

13. Complete q survey online 


2.94 

2.89 

2.91 

14. Submit qssignments online using a drop box 


2.64 

2.44 

2.54 

15. View my grades in the grade book 


2.88 

2.86 

2,87 


Mean 

2.78 

2.68 

2.73 

Part III Asynchronous Communication 

1 would feel confident to .... 





16. Send text-based e-mail to my instructor 


3.00 

2.80 

2.90* 

17. Send text-based e-mail to one or more students in my class 


2.88 

2.66 

2.76* 

18. Send voice e-mail to my instructor 


1.64 

0.89 

1.25* 

19. Send voice e-mail to one or more students 


1.61 

0.86 

1.22* 

20. Post text messages in the discussion group 


2.97 

2.63 

2.79* 

21. Reply to the text messages in the discussion group 


2.88 

2.56 

2.71* 

22. Create a new thread in the discussion group 


2.82 

2.14 

2,46* 

23. Download attachments from the messages in the discussion group 


2.88 

2.53 

2.70* 

24. Attach files to my messages in the discussion group 


2.85 

2.50 

2.67* 

25. Post voice messages to the voice board 


1.61 

0.83 

1.20* 

26. Reply to the voice board messages 


1.67 

1.03 

1.33* 

27. Import and export voice messages 


1.55 

0.83 

1.17* 

28. Create an audio Podcast 


1.09 

0.67 

0.87* 

29. Exchange files with my group members 


2.58 

2.17 

2.36* 


Mean 

2.29 

1.79 

2.02* 

Part IV Synchronous Communication 

1 would feel confident to .... 





30. Join a text-based chat session 

31. Read messages from one or more members in a synchronous 


2.58 

2.31 

2.43* 

text-based chat system 

32. Post or reply to a message in a synchronous text-based chat 


2.55 

2.17 

2.35* 

system (one-to-many interaction) 

33. Interact privately with one member of the synchronous 


2.55 

2.14 

2.33* 

text-based chat system (one-to-one interaction) 


2.55 

2.14 

2.33* 

34. View archived text-based chat sessions 

35. Join a virtual class session, such as Horizon Wimba or 


2.27 

2.00 

2.13 

Blackboard Virtual Classroom 


2.15 

1.56 

1.84* 

36. Use the Whiteboard tools in a virtual class session 


1.61 

1.25 

1.42 

36. Use the Whiteboard tools in a virtual class session 


1.61 

1.25 

1.42 

37. Join a breakout room in a virtual class session 


1.64 

1.08 

1.35* 

38. Display a Web browser from within a virtual class session 


1.67 

1.33 

1.49 

39. Ask questions to the moderator of the virtual class session 


2.00 

1.47 

1.72* 

40. Direct message with the other participants in the virtual class session 

41. Post my responses by selecting different options 


1.88 

1.42 

1.64* 

(e,g„ polling, hand raising) in the virtual class session 

42. Moderate a virtual class session (e.g., load presentations. 


1.73 

1.36 

1.54 

archive settings, grant user permissions) 


1.33 

1.22 

1.28 


Mean 

2.04 

1.65 

1.84* 


Table 1. Mean scores for LMSES before differential analysis (48 items). (Cont..) 
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Part V Advanced Tools 
I would feel confident to 


43. Post my reflection to a journal 


2.18 

2.14 

2.16 

44. Post my reflection to a blog 


2.15 

1.94 

2.04 

45. Comment on a blog posting 

46. Collaborate on web pages to add, expand, and change the 


2.18 

2.11 

2.14 

content (Wiki) 


2.00 

1.42 

1 . 70 * 

47. Read news publications using RSS feeds 


1.85 

1.34 

1 . 59 * 

48. Get context-sensitive help 


2.12 

1.58 

1 . 84 * 


Mean 

2.08 

1.76 

1.91* 


Table 1. Mean scores for LMSES before Differential Analysis (48 items). 


LMSES Functionality after differential analysis, 


Online Hybrid Overall Mean 


Part I Accessing Information 
I would feel confident to .... 


1. Log in to my course in the LMS 

2.88 

2.75 

2.81 

2. Read the text-based announcements posted by my instructor 

2.94 

2.72 

2.83* 

3. View my instructor's information, such as name, office hours, and office location 


2.78 

2.87* 

4, View the course documents online 

2.97 

2.78 

2.87* 

5. Access the links to the Web resources 

2.94 

2.83 

2.88 

6. View the feedback for the online test/quiz 

2.79 

2.75 

2.77 

7. Access the course calendar and tasks assigned 

2.88 

2.83 

2.85 

8, View my grades in the grade book 

2.88 

2.86 

2.87 

Mean 

2.91 

2.79 

2.84 

Part II Posting Information 




9. Take a test/quiz online 

2.67 

2.47 

2.57 

10. Send text-based e-mail to my instructor 

3.00 

2.80 

2.90* 

11. Post text messages in the discussion group 

2.97 

2.63 

2.79* 

12. Create a new thread in the discussion group 

2.82 

2.14 

2.46* 

Mean 

2.87 

2.51 

2.57* 

Part III File Management 




13. Submit assignments online using a drop box 

2.64 

2.44 

2.54 

14. Download attachments from the messages in the discussion group 

2.88 

2.53 

2.70* 

15. Exchange files with my group members 

2.58 

2.17 

2.36* 

Mean 

2.70 

2.38 

2.54 * 

Part IV Advanced Features 




16. Join a virtual class session, such as Horizon Wimba or 




Blackboard Virtual Classroom 

2.15 

1.56 

1.84* 

17. Use the Whiteboard tools in a virtual class session 

1.61 

1.25 

1.42 

18. Join a breakout room in a virtual class session 

1.64 

1.08 

1.35* 

19. Display a Web browser from within a virtual class session 

1.67 

1.33 

1.49 

20. Direct message with the other participants in the virtual class session 

21. Post my responses by selecting different options (e.g., Polling, 

1.88 

1.42 

1.64* 

hand raising) in the virtual class session 

22. Moderate a virtual class session (e.g., load presentations, archive settings. 

1.73 

1.36 

1.54 

grant user permissions) 

1.33 

1.22 

1.28 

23. Post my reflection to a blog 

2.15 

1.94 

2,04 

24. Collaborate on web pages to add, expand, and change the content (Wiki) 

2.00 

1.42 

1 . 70 * 

Mean 

1.80 

1.40 

1.55* 


Table 2. Mean scores for LMSES after Differential Analysis (24 items). 
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Student confidence for accessing the course content was 
(M=2.84), posting information (M=2.57), file management 
(M=2.54), and advanced features (M=1.55). The reliability 
of this administration was .92. Factor and post-hoc 
analysis were used to examine our a priori hypothesis that 
the scale contained four dimensions. 

Follow-up univariate analyses for the four categories 
revealed significant differences at the pc.01 level for 
three of the four categories, all indicating greater self- 
efficacy for students in the online course. Tukey post hoc 
analyses revealed significant differences on 12 of the 24 
items again, all favoring students in the online course. 

LMS Self-Efficacy and Course Performance 
Regression analysis was conducted to predict the effect 
of learner LMS self-efficacy on course performance. In this 
study, LMS self-efficacy of the hybrid learners accounted 
for a significant contribution to their course performance 
R2 = .09, F(1,35) = 3.26, pc,05, indicating students with 
higher LMS self-efficacy tended to have better course 
performance. Whereas, LMS self-efficacy of the online 
learners did not account for a significant contribution their 
course performance R2 = .04, F(1,32) = 1,16., p >.05. 
Discussion 

Perhaps the most interesting finding is the significant 
positive correlation of self-efficacy with course 
performance for the students in the hybrid course, despite 
reporting significantly lower self-efficacy than the students 
in the online course in three of the four categories 
measured. Neither group reported a relatively high level 
of self-efficacy. The highest reported self-efficacy value 
for either group, "Send text-based e-mail to my instructor," 
had a mean of 3.0 (Somewhat Confident) for online 
learners. This could mean there is a baseline 
competence with LMS use required for success, but once 
that level is perceived, greater self-efficacy with the 
system is not required. 

Furthermore, it is perplexing that the significant positive 
correlation occurred for the hybrid learners. It would seem 
that the use of the LMS as a supplement to face-to-face 
instruction would require less confidence with the system 
than in a course in which all content is delivered though 


the LMS. Other factors that may have influenced this 
finding could be discrepancies in the use of various tools 
between the courses or other differences in the learners. 
Hybrid learners had the option to enroll in the fully online 
version of the course, but self-selected into the hybrid 
version. This may be due in part to their perceived lower 
self-efficacy with the delivery system. Clearly, more 
investigation is required. 

Finally, the only LMSES category that did not yield a 
significant difference was Tests and Grades when 
grouped into five categories (M=2.73) and it was 
Accessing Information (M=2.84) when grouped into four 
categories. The authors suggest this finding is an 
indication of the predominant use of a LMS throughout 
each student's experience. As suggested by Bandura 
(1997), the formation of self-efficacy beliefs is based 
primarily on reflection and interpretation of past 
performance. If this is the case, it is unfortunate that the 
vast array of learning support features of a contemporary 
LMS is not utilized, 

Conclusion 

Although LMS has been widely used in higher education, 
this study indicated that many LMC features are not fully 
utilized and a large portion of students are still lack of 
confidence with the system. Future studies should focus 
on identifying factor that influence the level of technology 
self-efficacy and help students gain more confidence. 
More studies could be designed to investigate the 
human-computer interaction of LMS. Studies are also 
needed to examine other factors influencing course 
format selection. 
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